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Abstract 


Open  Systems  and  Evolutionary  Acquisition  are  two  recent  innovations 
designed  to  improve  program  performance  with  flexibility.  The  full  potential  of  these 
approaches  has  not  been  captured,  partially  because  of  integration  challenges 
during  implementation.  The  current  work  investigates  the  impacts  of  open  systems 
and  evolutionary  acquisition  on  DoD  development  programs.  Changes  required  to 
use  both  Open  Systems  and  Evolutionary  Acquisition  are  used  to  identify  and 
describe  impacts  of  implementation  on  program  process  and  management.  A 
dynamic  simulation  model  of  a  program  using  both  Evolutionary  Acquisition  and 
Open  Systems  is  described  and  used  to  map  the  impacts.  Simulation  results 
generally  support  previously  suggested  impacts  and  provide  a  possible  explanation 
for  changes  in  program  performance.  Implications  for  practice  relate  to  changes  in 
the  types  and  timing  of  risk  and  a  potential  trading  of  design  obsolescence  risk  for 
standards  obsolescence  risk. 
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Introduction 


System  interoperability  and  the  incorporation  of  evolving  technologies  in 
major  DoD  systems  are  two  important  acquisition  challenges  that  the  military  faces 
in  preparing  the  warfighter  to  meet  current  and  future  capability  demands.  The  use 
of  legacy  and  other  weapon  platforms,  joint  service  solutions,  the  information  and 
communication  needs  of  Network-centric  Systems  (NCS),  and  coordination  with 
allies  in  joint  operations  each  require  the  development  of  weapon  systems  that  can 
operate  across  system,  platform,  and  systems-of-systems  boundaries.  Past  DoD 
acquisition  approaches  have  not  fully  provided  the  interoperability  needed  to  meet 
these  demands.  The  continued,  and  in  some  cases  accelerating,  evolution  of 
technologies  creates  new  challenges  that  are  difficult  to  forecast  and  require  fast 
acquisition  response.  Integrated  human-computer  decision-making  tools,  advanced 
materials,  NCS  tools,  and  nano-level  structures  are  examples  of  evolving 
technologies  that  present  challenges  and  potential  solutions  that  must  be  integrated 
by  defense  acquisition  programs. 

Open  systems  (OSJTF,  2004,  September)  and  evolutionary  acquisition  (DoD, 
2004,  November,  section  4.4.1)  are  two  relatively  recent  DoD  acquisition  initiatives 
that  seek  to  address  system  interoperability  and  technology  evolution  challenges 
and  that  help  the  DoD  meet  current  and  future  capability  needs.  An  open  systems 
(OS)  approach  and  evolutionary  acquisition  (EA)  share  several  high-level  objectives. 
Both  approaches  seek  to  improve  performance  over  the  system’s  lifetime  and 
reduce  acquisition  cycle-time.  Both  approaches  also  attempt  to  improve  system 
performance  via  flexibility  for  the  integration  of  new  technologies  and  information  into 
systems  as  they  evolve.  The  open  systems  approach  facilitates  upgrades  through 
modularity.  EA  does  this  by  multiple  product  releases  and  deliberate  deferral  of 
some  functionality — allowing  technologies  and  requirements  to  evolve  and  mature. 
Both  OS  and  EA  seek  to  reduce  acquisition  cycle-time  to  provide  currently  available 
functionality.  OS  provide  a  means  of  incorporating  current  and  future  functionality. 
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and  evolutionary  acquisition  limits  the  scope  of  develop  blocks  to  only  the 
technologies  and  capabilities  that  are  attainable  in  the  near  future. 

Open  systems  and  evolutionary  acquisition  share  at  least  two  important 
implementation  approaches.  First,  both  OS  and  EA  incorporate  flexibility  into 
acquisition  to  manage  uncertainty  in  technology.  Open  Systems  build  flexibility  into 
development  products  with  modular  design  and  standardized  key  interfaces. 
Evolutionary  acquisition  builds  flexibility  into  development  processes  through  the 
design  of  incremental  capability  blocks.  These  flexibilities  create  options  that 
potentially  increase  system  performance,  reduce  cost,  or  both,  by  allowing 
technological  uncertainties  to  partially  resolve  before  important  development 
decisions  are  made.  Second,  both  OS  and  EA  place  emphasis  upon  interfaces  to 
address  interoperability.  Within  an  evolutionary  approach,  interface  management  is 
critical  to  successfully  integrating  designs  across  development  blocks.  This  need 
increases  for  systems  with  interfaces  across  platforms  or  systems-of-systems.  In 
contrast  to  these  challenges,  an  OS  approach  focuses  on  explicitly  identifying  and 
managing  key  interfaces  that  can  benefit  from  modular  design  and  open  systems  as 
a  means  of  improving  interoperability. 

The  evolutionary  acquisition  challenge  and  the  open  systems  method  suggest 
that  the  two  acquisition  approaches  must  be  integrated  and  may  be  synergistic.  But 
the  complexity  of  the  processes  and  the  requirements  of  the  two  approaches  make 
their  integration,  synergy,  and  successful  implementation  anything  but  obvious,  easy 
or  certain.  The  requirements  of  the  approaches  have  been  largely  identified,  and 
some  of  the  changes  required  in  programs  for  the  use  of  EA  and  OS  together  have 
been  identified.  But  a  focused  study  of  the  impacts  of  integrating  open  systems  and 
evolutionary  acquisition  is  needed  both  to  identify  the  impacts  on  development 
processes  and  to  point  to  potential  program  design  and  management  actions  in 
order  to  exploit  their  potential.  How  does  the  use  of  evolutionary  acquisition  and 
open  systems  together  impact  a  system’s  development  processes  and 
management?  How  do  those  impacts  affect  acquisition  program  performance? 
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The  current  work  partially  addresses  these  issues  as  follows.  The  researchers 
review  evolutionary  acquisition  and  open  systems  approaches  through  the  lens  of 
their  influence  on  program  processes  and  management.  The  researchers  then  use 
the  required  program  changes  identified  in  the  existing  literature  to  describe 
challenges  to  integrating  the  approaches  and  to  describe  specific  influences  on 
program  management.  After  describing  the  modeling  approach  used  here  and  the 
simulation  model  of  an  acquisition  program,  the  researchers  map  the  specific 
influences  into  changes  in  model  variables.  They  then  use  the  results  of  simulations 
of  the  evolutionary  acquisition  program  without  and  with  open  systems  as  a  basis  for 
a  discussion  of  both  the  needs  for  successful  programs  that  use  both  approaches, 
as  well  as  the  use  of  simulation  modeling  as  a  tool  for  investigating  these 
acquisition-implementation  issues.  The  paper  closes  with  recommendations  for 
future  work. 

Evolutionary  Acquisition 

In  the  year  2000,  the  Defense  Department  promulgated  the  term  “evolutionary 
acquisition”  (EA)  in  its  policy  documents  governing  the  strategy  for  acquisition  of 
materiel  and  mandated  such  strategies  be  used  as  the  preferred  approach  to 
procurement  (USD(AT&L),  2000,  October  23).  Later  elaborated  as  spiral  and 
incremental  strategies,  these  approaches  contrast  to  others  that  are  based  on  more 
serial,  sequential  or  singular  efforts  to  arrive  at  a  product  solution.  The  latter  are 
often  termed  as:  single-step-to-full-capability,  grand  design,  big  bang,  technological 
leap,  waterfall,  rational-comprehensive,  and  the  unified  development  method 
(Forsberg,  Mooz,  &  Cotterman,  2005,  p.  354).  The  overarching  goals  and  principles 
of  the  DoD’s  evolutionary  acquisition  are  to  ensure  that  the  Defense  Acquisition 
System  provides  useful  military  capability  to  the  operational  user  as  rapidly  as 
possible,  and  such  strategies  shall  be  the  preferred  approach  to  satisfying 
operational  needs.  Evolutionary  acquisition  strategies  define,  develop,  and 
produce/deploy  an  initial,  militarily  useful  capability  ("Block  I")  based  upon  proven 
technology,  time-phased  requirements,  projected  threat  assessments,  and 
demonstrated  manufacturing  capabilities.  They  also  plan  for  subsequent 
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development  and  production/deployment  of  increments  beyond  the  initial  capability 
over  time  (Blocks  II,  III,  and  beyond)  (USD(AT&L),  2000,  October  23).  Figure  1 
shows  the  conceptual  difference  between  a  traditional  single-step-to-capacity 
acquisition  process  and  an  evolutionary  acquisition  process  with  two  development 
blocks,  as  described  in  the  1996  and  2003  versions  of  DoD  5000  series. 


1996  and  2003  DoD  5000  Models 
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Figure  1.  Comparison  of  Traditional  Single-step-to-capacity 
and  Evolutionary  Acquisition  Approaches 

(Dillard,  2005) 

The  policy  for  evolutionary  acquisition  was  aimed  at  improving  all  parameters 
of  program  success,  but  clearly  and  explicitly,  its  single  most  important  objective  was 
to  reduce  long  product  cycle-times  to  deliver  operationally  useful  equipment.  Figure 
1  illustrates  the  hypothetical  earlier  start  of  production  and  the  overlapping 
development  blocks  that  are  characteristic  of  evolutionary  acquisition.  The  authors, 
in  their  previous  work  (Dillard  &  Ford,  2007)  investigated  implementation  challenges 
of  evolutionary  acquisition  using  the  same  approach  that  we  are  using  in  the  current 
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work.  We  found,  in  part,  that  an  evolutionary  development  approach  significantly 
increases  the  number  of  development  phases  and  activities  that  must  be  managed 
and  coordinated  at  any  given  time  over  that  required  for  single-block  development. 
This,  consequently,  increases  the  organizational  project  management  resource 
needs  for  successful  acquisition  over  those  necessary  for  single-block  projects. 
Using  open  systems  with  an  evolutionary  approach  may  or  may  not  accentuate 
these  challenges. 


Open  Systems  in  DoD  Acquisition 

Open  Systems  were  made  a  part  of  DoD  acquisition  in  DoD  5000.1  (Under 
Secretary  of  Defense  (AT&L),  2003,  May  12a),  which  says  “a  modular  open  systems 
approach  shall  be  employed  where  feasible”  (p.  7).  A  subsequent  memorandum 
(Under  Secretary  of  Defense  (AT&L),  July  7,  2004)  clarified  the  central  role  of  OS  in 
acquisition  by  saying  the  approach  is  “an  integral  part  of  the  toolset  that  will  help 
DoD  achieve  its  goal  of  providing  the  joint  combat  capabilities  required  in  the  21st 
century,  including  supporting  and  evolving  these  capabilities  over  their  total  life- 
cycle”  (p.  8).  The  Open  Systems  Joint  Task  Force  (OSJTF)  leads  the  DoD  OS  effort 
(OSJTF,  2004,  September).  Several  terms  defined  in  that  guide  are  relevant  to  and 
used  in  the  current  work,  including: 

■  Open  architecture:  An  architecture  that  employs  open  standards  for 
key  interfaces  within  a  system. 

■  Open  Standards:  Standards  that  are  widely  used,  readily  available, 
consensus-based,  published  and  maintained  by  recognized  industry 
standards  organizations  (versus  “closed,”  which  are  not). 

■  Open  system:  A  system  that  employs  modular  design,  uses  widely 
supported  and  consensus-based  standards  for  its  key  interfaces,  and 
has  been  subjected  to  successful  validation  and  verification  tests  to 
ensure  the  openness  of  its  key  interfaces. 

■  Open  systems  environment  (OSE):  A  comprehensive  set  of 
interfaces,  services,  and  supporting  formats,  plus  aspects  of 
interoperability  of  application,  as  specified  by  Information  Technology 
(IT)  standards  and  profiles.  An  OSE  enables  information  systems  to  be 
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developed,  operated,  and  maintained  independent  of  application- 
specific  technical  solutions  or  vendor  products. 

An  open  systems  approach  uses  the  concepts  of  key  versus  non-key 
interfaces  and  open  versus  closed  interfaces,  as  defined  above,  to  build  flexibility 
into  programs.  Figure  2  illustrates  potential  locations  of  these  interfaces  in  a 
conceptual  system  with  modular  subsystems/components.  The  centrality  of  these 
concepts  to  the  open  systems  approach  greatly  increases  the  importance  of  the 
intended  and  unintended  impacts  of  a  shift  away  from  the  traditional  focus  on 
customized  designs  to  integration  through  open  interfaces. 


Figure  2.  Types  of  Systems  Interfaces 

(OSJFT,  2004) 
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Challenges  of  Integrating  Evolutionary  Acquisition 
and  Open  Systems 


Program  managers  using  open  systems  and  evolutionary  acquisition  in  an 
integrated  fashion  may  be  able  to  achieve  interoperability  and  insert  evolving 
technologies  better  than  using  either  approach  alone.  But,  despite  their  potential,  the 
combination  of  OS  and  EA  has  not  yet  been  fully  developed  or  implemented  in  DoD 
acquisition.  This  is  perceived  to  be  largely  because  the  issues  related  to  their 
implementation  have  not  been  completely  identified  or  resolved.  This  incomplete 
resolution  of  the  implementation  of  open  systems  and  evolutionary  acquisition 
makes  understanding  their  interactions  and  the  impacts  of  those  interactions  on 
acquisition  programs  difficult. 

The  adoption  and  use  of  open  systems  in  DoD  acquisition  requires  several 
different  activities  that  impact  the  acquisition  process  in  different  ways.  Meyers  and 
Oberndorf  (2001)  identify  some  of  these  activities.  We  describe  the  most  important 
activities  identified  by  Meyers  and  Oberndorf  with  our  assessment  of  their  impacts 
on  the  evolutionary  acquisition  process: 

1.  Build  a  baseline  of  standards  and  commercial-off-the-shelf 
(COTS)  products.  This  change  increases  the  scope  of  the  Block  1 
requirements  phase  and  early  design  (pre-system  acquisition)  to 
describe  the  requirements  in  terms  of  standards. 

2.  Build  a  high-level  model  of  the  system  for  use  in  applying  the 
open  systems  approach.  This  change  increases  the  scope  of  early 
design  in  Block  1 . 

3.  Document  the  open  architecture  in  a  way  that  shows  the 
evaluation  of  alternative  architectures,  identifies  components, 
technologies,  etc.  This  change  increases  the  scope  of  the  early 
design  activities  and  advanced  development  phases  in  all  Blocks. 

4.  Coordinate  standards  and  establish  liaisons  with  standards 
bodies  and  users.  This  change  increases  the  scope  of  all  phases  in 
all  blocks  because  it  is  an  on-going  process. 
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5.  Implement  the  use  of  the  selected  standards  in  the  development 
process.  This  change  decreases  the  scope  of  the  advanced 
development  phase  in  all  blocks  due  to  component  design  activities 
being  replaced  with  component  selection. 

6.  Integrate  components  into  the  product  and  test  the  integrated 
system.  This  change  increases  problems/rework  in  advanced 
development  and  manufacturing  phases  of  all  blocks. 

Hanratty,  Lightsey,  and  Larson  (1999)  also  investigated  the  use  of  open 
systems  in  acquisition.  They  describe  the  impacts  of  OS  on  acquisition  as  a  shift 
away  from  design  (which,  in  OS,  is  done  by  the  broader  commercial  market)  to  an 
integration  of  elements  into  products  (which,  in  OS,  is  increasingly  done  with 
elements  that  were  not  developed  specifically  for  the  DoD).  Hanratty,  Leghtsey,  and 
Larson  identified  several  areas  of  open  systems  design  that  pose  risks,  which  we 
describe  with  our  assessment  of  the  primary  impacts  of  OS  on  evolutionary 
acquisition  processes. 

1.  Slower  integration  and  testing  of  standards-based  elements  into 
products.  This  change  delays  the  discovery  of  integration  problems 
until  later  in  projects. 

2.  Reduced  DoD  control  over  standards.  This  change  increases  the 
number  and  size  of  design  problems  due  to  faster  evolution  of  the 
standard  used  in  the  product. 

3.  Increased  standards-selection  risk  due  to  evolution  of  standards 
and  the  possibility  that  standards  will  not  endure.  This  change 
increases  the  number  and  size  of  design  problems  due  to  the 
possibility  that  the  selected  standard  will  not  endure,  and  increases 
testing  and  integration  (regardless  of  whether  problems  are  discovered 
or  not)  due  to  more  frequent  changes  in  standards. 

4.  Increased  standard  change  risk — knowing  when  to  shift  from  one 
standard  to  another.  This  change  increases  testing  and  integration 
(regardless  of  whether  problems  are  discovered  or  not)  due  to  more 
frequent  changes  in  standards.  It  also  increases  the  number  and  size 
of  integration  problems  that  need  to  be  discovered  and  resolved  due  to 
the  need  to  change  to  the  new  standard  more  often  and  the  possibility 
of  changing  too  early,  too  late,  or  to  the  wrong  standard  if  more  than 
one  are  available  (e.g.,  competing  for  market  dominance). 
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5.  Increased  and  continuous  testing  requirements  due  to  the  need  to 
integrate  evoiving  commerciai  and  non-deveiopmentai  items  into 
systems.  This  change  increases  testing  and  integration  (regardless  of 
whether  problems  are  discovered  or  not)  due  to  more  frequent 
component  redesigns. 

6.  Development  of  support  concepts  early  in  the  acquisition  cycle — 
causing  increased  standards-selection  risk  due  to  large  amounts 
of  information  needed  about  currently  available  standards.  This 
change  increases  standards  research  and  planning  early  in  acquisition, 
which  would  include  increased  interface  design  and  management. 

7.  Reduced  control  over  detailed  component  design  due  to  design 
by  industry  based  on  industry-controlled  standards.  This  change 
increases  the  number  and  size  of  integration  problems  due  to 
component  designs  that  do  not  exactly  match  product  needs. 

These  specific  influences  pose  significant  individual  challenges.  However, 
they  might  also  interact  in  ways  that  are  difficult  to  predict  or  immediately  recognize 
and  address.  In  the  Model  Use  section,  we  describe  how  we  mapped  these 
influences  onto  specific  parts  of  an  acquisition  process  to  better  understand  how 
they  impact  program  performance. 
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The  Research  Approach 


Evolutionary  acquisition  and  open  systems  approaches  combine  to  create  a 
complex  set  of  development  processes  that  evolve  over  time.  An  improved 
understanding  of  these  processes  and  their  management  is  available  through  formal 
modeling  of  the  most  important  components  and  relationships  that  drive  system 
performance  and  risk.  Due  to  the  number  and  complexity  of  the  components  and 
their  relationships,  the  formal  model  structure  and  rigor  of  calculations  can  simulate 
and  forecast  performance  and  risk  better  than  informal  tacit  predictions  by  humans. 
Therefore,  we  applied  a  computational  experimentation  approach  to  investigating 
evolutionary  acquisition  and  open  systems  projects,  integrating  theory  and  practice 
in  a  computational  tool  that  allows  controlled  experimentation  through  simulation. 

The  current  work  reflects  project,  product  development,  and  management  theories. 

The  system  dynamics  methodology  was  applied  to  model  a  DoD  acquisition 
project  with  evolutionary  processes  and  open  systems.  System  dynamics  uses  a 
computational  experimentation  approach  to  understanding  and  improving 
dynamically  complex  systems.  The  system  dynamics  perspective  focuses  on  the 
roles  of  accumulations  and  flows,  feedback,  and  nonlinear  relationships  in 
managerial  control.  The  methodology’s  ability  to  model  many  diverse  system 
components  (e.g.,  work,  people,  money),  processes  (e.g.,  design,  technology 
development,  quality  assurance),  and  managerial  decision-making  and  actions  (e.g., 
forecasting,  resource  allocation)  makes  it  useful  for  investigating  acquisition  projects. 
Forrester  (1961)  develops  the  methodology's  philosophy,  and  Sterman  (2000) 
specifies  the  modeling  process  with  examples  and  describes  numerous  applications. 
When  applied  to  development  projects,  system  dynamics  focuses  on  how 
performance  evolves  in  response  to  interactions  among  development  strategy  (e.g., 
evolutionary  development  vs.  traditional),  managerial  decision-making  (e.g.,  scope 
developed  in  specific  blocks),  and  development  processes  (e.g.,  concurrence). 
System  dynamics  is  considered  appropriate  for  modeling  acquisition  projects 
because  of  its  ability  to  explicitly  model  critical  aspects  of  development  projects 
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(Ford  &  Sterman,  1998;  Cooper,  1993a,b,c;  Cooper  &  Mullen,  1993;  Cooper,  1994). 
System  dynamics  has  been  successfully  applied  to  a  variety  of  project  management 
issues,  including  prediction/discovery  of  failures  in  project  fast-track  implementation 
(Ford  &  Sterman,  2003b),  poor  schedule  performance  (Abdel-Flamid  1988),  and  the 
impacts  of  changes  (Rodriguez  &  Williams,  1997;  Cooper,  1980)  and  concealing 
rework  requirements  (Ford  &  Sterman,  2003a)  on  project  performance.  See  Lyneis 
and  Ford  (2007)  for  a  review  of  the  application  of  system  dynamics  to  projects. 

The  simulation  model  used  here  is  based  on  previously  developed  system 
dynamics  models  of  product  development  in  several  industries  that  have  been 
developed  and  tested  over  several  decades,  as  described  and  referenced  below. 
Therefore,  the  model  is  founded  on  well-established  and  tested  components. 
Previous  models  have  developed  structures  for  many  components  and  aspects  of 
acquisition.  However,  previous  models  have  not  been  used  to  investigate  the 
integration  of  EA  and  OS  in  acquisition  projects.  The  current  model  was  originally 
developed  to  investigate  EA  and  is  described  in  detail  by  Dillard  and  Ford  (2007). 
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A  Conceptual  Model  of  an  Evolutionary  Acquisition 
Program 


The  model  structure  reflects  the  structure  of  development  work  moving 
through  the  separate  development  blocks  of  an  acquisition  project.  In  the  model,  four 
types  of  work  flow  through  each  block  of  an  acquisition  project:  the  development  of 
requirements,  the  development  of  technologies,  the  design  of  product  components, 
and  the  manufacture  of  products.  Within  a  development  block,  each  type  of  work 
flows  through  a  development  phase  that  completes  a  critical  aspect  of  the  project:  1) 
develop  requirements,  2)  develop  technologies,  3)  design  product  components 
(advanced  development),  and  4)  manufacture  products.  The  exception  is 
requirements,  which  also  measures  progress  through  the  final  phase,  5)  conduct 
user  product  testing.  Development  phases  and  information  flows  in  a  single  block,  as 
depicted  in  the  model,  are  shown  in  Figure  3.  Arrows  between  phases  indicate 
primary  information  flows.  The  start  of  all  phases  (except  the  development  of 
requirements)  is  constrained  by  the  completion  of  previous  (“upstream”)  phases.  The 
completion  of  some  requirements  allows  the  start  of  technology  development, 
reflecting  the  concurrent  nature  of  this  portion  of  acquisition.  Both  requirements 
development  and  technology  development  must  be  completed  for  Advanced 
Development  to  begin.  The  completion  of  Advanced  Development  allows 
manufacturing  to  begin.  When  some  products  have  been  manufactured,  they  are 
shipped  to  users  for  readiness  testing.  Figure  3  also  identifies  the  five  major  reviews 
within  a  single  acquisition  block  (A,  B,  Design  Readiness  Review,  C,  and  Full-rate 
Production)  at  their  approximate  times  during  a  project.  These  reviews  are 
necessary,  but  are  “off-core”  activities  that  add  work  beyond  that  needed  to 
complete  the  basic  products  of  each  phase  (requirements,  technologies,  designs, 
products,  and  readiness  for  use  confirmation). 
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Time  Periods 
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Figure  3.  Information  Flows  in  a  Single-block  Acquisition  Project 

Figure  4  depicts  an  acquisition  project  with  multiple  iterations  or  blocks.  The 
first  block  is  the  same  as  Figure  3  above.  Subsequent  blocks  have  the  same  basic 
information  flow,  but  can  also  be  delayed  by  the  completion  of  phases  in  previous 
blocks  or  constrained  by  the  lack  of  progress  in  their  own  block.  Importantly,  in 
addition  to  the  flow  of  information  downstream  through  phases  (black  arrows  in 
Figure  4),  multiple  iteration  acquisition  also  provides  opportunities  for  information  to 
flow  upstream,  such  as  from  User  Product  Testing  in  an  earlier  iteration  to  Develop 
Requirements  or  Advanced  Development  in  a  subsequent  iteration  (red  vertical 
arrows  in  Figure  4). 
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Figure  4.  Information  Flows  in  a  Three-block  Acquisition  Project 
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A  Formal  Simulation  Model  of  an  Evolutionary 
Acquisition  Program 


The  conceptual  model  described  above  was  used  to  build  a  formal  computer 
simulation  model  of  an  acquisition  program  that  can  reflect  evolutionary  acquisition 
and  the  use  of  open  systems.  See  Dillard  and  Ford  (2007)  for  details.  The  simulation 
model  is  a  system  of  nonlinear  differential  equations.  Each  phase  is  represented  by 
a  generic  structure,  which  is  parameterized  to  reflect  a  specific  phase  of 
development. 

Project  performance  is  measured  in  three  dimensions:  schedule,  cost,  and 
product-performance  risk.  Schedule  performance  is  measured  in  the  time  required 
for  developers  and  users  to  produce,  test  and  approve  a  given  number  or  fraction  of 
requirements.  Cost  is  measured  in  dollars  based  on  the  size  of  direct  and  indirect 
work  forces  and  the  duration  of  phases  and  blocks.  Product-performance  risk  is 
measured  by  the  average  percent  of  the  requirements  provided  (approved  by  users) 
at  any  given  time.  This  average  reflects  the  combination  of  multiple  requirements.  All 
the  requirements  can  be  considered  met  completely  when  the  average  percent  of 
the  requirements  provided  is  100%  for  a  development  block. 

The  formal  model  was  calibrated  to  the  Javelin  project  described  by  Dillard 
and  Ford  (2007)  based  on  data  collected  from  a  manager  on  the  project  (the  second 
author)  and  performance  data  (e.g.,  schedule  and  costs)  on  the  project.  The  model 
was  tested  with  the  three  types  of  tests  of  system  dynamics  models  suggested  by 
Forrester  and  Senge  (1980):  structural  similarity  to  the  actual  system,  reasonable 
behavior  over  a  wide  range  of  input  values,  and  behavior  similarity  to  actual 
systems.  This  model  was  found  to  be  useful  for  investigating  the  impacts  of  OS  and 
EA  on  acquisition  projects. 
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Model  Use 


To  investigate  the  impacts  of  open  systems  on  evolutionary  acquisition,  we 
simulated  a  project  similar  to  the  Javelin  project  twice:  first  as  if  the  project  did  not 
use  open  systems  and  then  as  if  the  project  used  an  open  systems  approach.  We 
then  compared  the  behavior  and  project  performance.  The  program  base-case 
model  and  simulation  described  in  Dillard  and  Ford  (2007)  reflects  an  evolutionary 
acquisition  program  that  does  not  include  open  systems  impacts.  To  add  the  impacts 
of  open  systems  to  the  model,  we  first  mapped  the  identified  impacts  based  on 
Meyers  and  Oberndorf  (2001)  onto  model  variables  as  follows  (Table  1): 


Table  1.  Impacts  of  Open  Systems  on  Evolutionary  Acquisition  Due  to 
Changes  Suggested  by  Meyers  and  Oberndorf  (2001) 


Change  Required  by 
Open  Systems 

1)  Build  standards  &  COTS 
for  program  use 

2)  Build  high-level  model 
with  open  systems 

3)  Document  use  of  OS 

4)  Coordinate  standards 

5)  Implement  OS 


6)  Integrate  components 


Impact  on  Evolutionary  Acquisition  Processes 

Increases  Requirements  scope  in  Blocki 
Increases  Technology  Development  scope  in  Block  1 
Increases  Technology  Development  scope  in  Block  1 

Increases  Technology  Development  scope  in  all 
blocks 

Increases  scope  of  all  phases  in  all  blocks 

Decreases  Advanced  Development  scope  in  all 
blocks 

Fewer  Advanced  Development  design  problems  in  all 
blocks 

More  Advanced  Development  integration  problems  in 
all  blocks 

More  Manufacturing  integration  problems  in  all  blocks 


We  also  mapped  the  impacts  of  required  changes  to  acquisition  projects 
identified  by  Flanratty,  Lightsey,  and  Larson  (1999)  onto  model  variables  as  follows 
(Table  2): 
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Table  2.  Impacts  of  Open  Systems  on  Evolutionary  Acquisition  Due  to 
Changes  Suggested  by  Hanratty,  Lightsey,  and  Larson  (1999) 


Change  Required  by  Open  Systems 

7)  Slower  integration  and  testing 


8)  Track  and  change  with  evolving 
standards 

9)  Increase  testing  to  discover  increased 
integration  problems 

10)  Build  support  system  (OSE) 


Impact  on  Evolutionary  Acquisition 
Processes 

a1)  Reduces  problem  discovery  in 
Technology  Development  and 
Advanced  Development  phases  in  all 
blocks 

a2)  Increases  problem  discovery  in 
Manufacturing  phases  in  all  blocks 

b1)  Decreases  problem  discovery  in 
earlier  blocks  (all  phases  except 
Requirements) 

b2)  Increases  problem  discovery  in  later 
blocks  (all  phases  except 
Requirements) 

More  problems  in  Advanced 
Development  and  Manufacturing  phases 
in  later  blocks 

Increases  scope  in  Technology 
Development  and  Advanced 
Development  phases  in  all  blocks 

Increases  scope  in  Technology 
Development,  Advanced  Development, 
and  Manufacturing  phases  in  all  blocks 

Increases  scope  in  Requirements  phase 
in  Block  1 


Several  of  the  changes  above  impact  the  same  portions  of  an  evolutionary 
process,  sometimes  in  the  same  directions  and  sometimes  in  opposite  directions. 
Therefore,  we  regrouped  the  impacts  (Table  3)  according  to  model  variables  that 
describe  specific  program  blocks  and  development  phase  (e.g.,  scope  of  work  in 
Block  1,  Requirements  Phase).  The  three  variables  found  to  best  describe  the 
impacts  of  open  systems  on  evolutionary  acquisition  programs  are  the  scope  of 
work,  rework  fraction,  and  quality  assurance  (QA)  effectiveness.  In  the  table  below 
and  within  the  model,  the  scope  represents  the  work  that  must  be  completed  in  a 
development  phase.  The  Rework  Fraction  reflects  the  number  of  problems  that  are 
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created  in  a  development  phase.  The  QA  effectiveness  reflects  the  difficulty  of 
discovering  problems  to  be  resolved.  The  unit  of  measure  of  change  was  chosen  as 
the  percent  change  from  the  base  case  that  the  use  of  open  systems  would  cause. 
This  normalizes  impacts  for  different  phases  (e.g.,  a  change  of  10  to  a  phase  with  a 
scope  of  50  is  very  large  compared  to  the  same  change  to  a  phase  with  a  scope  of 
5,000)  and  facilitates  assessment  of  the  changes.  No  known  data  is  available  to 
complete  Table  3  based  on  an  actual  acquisition  program.  However,  order  of 
magnitude  estimates  that  are  in  a  reasonable  rank  order  of  size  are  adequate 
because  of  the  preliminary  nature  of  the  study.  The  net  changes  of  all  the  specific 
influences  are  summarized  in  Table  3.  See  Appendix  A  for  a  more  detailed 
description  of  the  estimates. 


Table  3.  Estimated  Changes  in  Evoiutionary  Acquisition  Processes  to 

Refiect  Open  Systems 


Program  Biock  and  Phase 

Scope  of 
Work 

Rework 

Fraction 

QA 

Effectiveness 

DEVELOPMENT  BLOCK  1 

Requirements 

+7 

0 

0 

Develop  Technology 

-15 

0 

-10 

Advanced  Development 

-17 

-5 

-10 

Manufacturing 

+2 

+5 

+5 

Testing  by  Users 

+1 

0 

-5 

Net  Change  from  Base  Case 

-22% 

0% 

-20% 

DEVELOPMENT  BLOCK  2 

Requirements 

+1 

0 

0 

Develop  Technology 

-16 

0 

-5 

Advanced  Development 

-17 

0 

-5 

Manufacturing 

+2 

+  10 

+  10 

Testing  by  Users 

+  1 

0 

0 

Net  Change  from  Base  Case 

-29% 

+10% 

0% 

DEVELOPMENT  BLOCK  3 

Requirements 

+1 

0 

0 

Develop  Technology 

-16 

0 

0 

Advanced  Development 

-17 

+5 

0 

Manufacturing 

+2 

+  15 

+  15 

Testing  by  Users 

+1 

0 

+5 

Net  Change  from  Base  Case 

-29 

+20 

+20 
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Simulation  Results 


Figure  5  shows  a  plot  of  the  simulated  percent  of  project  requirements 
provided  to  users  by  the  acquisition  program  without  open  systems  (Line  1)  and  with 
open  systems  (Line  2).  The  simulated  program  has  three  development  blocks,  and 
the  simulation  clearly  shows  the  evolutionary  acquisition  nature  of  the  program — with 
three  increases  in  requirements  provided  as  each  development  block  is  completed. 
The  simulation  also  shows  that  the  program  with  open  systems  provides  as  many  or 
more  requirements  at  any  point  in  time  than  the  program  without  open  systems.  This 
supports  the  open  systems  approach’s  claim  that  it  can  facilitate  providing  more 
requirements  faster. 


Requirements 
Provided  to 
Users 

(percent  of  all 
project 

requirements) 


Time  (Week) 


Figure  5.  Requirement  Fulfillment  with  Evolutionary  Acquisition 
without  (Line  1)  and  with  (Line  2)  Open  Systems 

In  addition  to  supporting  the  potential  gains  available  through  evolutionary 
acquisition  and  open  systems,  the  simulation  describes  the  interaction  of 
evolutionary  acquisition  and  open  systems  in  more  detail,  providing  the  opportunity 
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for  improved  understanding.  The  simulation  shows  that  the  improvement  in  time-to- 
requirement  increases  with  each  block,  indicating  that  open  systems  can  improve 
this  dimension  of  program  performance  during  multiple  development  blocks.  An 
open  systems  approach  may  leverage  its  benefits  when  used  with 
evolutionary  acquisition  through  repeated  capture  of  benefits  generated  in 
early  development  blocks  in  subsequent  development  blocks.  If  an  OS 
approach  is  implemented  with  EA,  programs  may  be  able  to  reap  the  benefits  first 
achieved  in  earlier  blocks  in  subsequent  downstream  blocks,  effectively  benefitting 
more  than  once  for  the  open  systems  work  done  early. 

However,  time  to  delivery  of  requirements  is  only  one  measure  of  program 
performance.  Cost  is  another  important  performance  measure.  The  simulated 
program  without  open  systems  costs  $5.39  million  through  complete  release  to 
users  and  the  program  with  open  systems  costs  $3.84  million  through  complete 
release  to  users. \  Reduced  costs  are  an  established  potential  benefit  of  using  open 
systems,  largely  through  reduced  design  scope.  This  is  the  case  in  the  model,  in 
which  a  significant  reduction  in  design  scope  is  assumed  to  be  a  fundamental  impact 
of  using  open  systems.  However,  the  simulation  points  out  an  additional  potential 
cost  benefit  of  using  open  systems.  Shorter  programs  tend  to  cost  less  (all  other 
things  held  equal).  Therefore,  open  systems  can  improve  cost  performance  by 
interacting  with  evolutionary  acquisition  to  enhance  the  schedule  performance 
available  through  evolutionary  acquisition  alone. 

A  third  important  performance  measure  is  the  quality  of  the  developed 
product.  Less-than-desired  quality  can  be  caused  by  many  things,  including  not  or 
partially  fulfilling  requirements,  design  errors  that  reduce  product  performance  or 
increase  operations  or  maintenance  costs,  and  integration  errors  that  make  future 
upgrades  difficult,  slow,  or  expensive.  Design  and  integration  errors  are  particularly 
important  in  the  current  work  because  of  their  central  role  in  open  systems. 
Acquisition  program  changes  required  by  open  systems  clearly  alter  the  nature. 


^  Actual  costs  may  be  significantly  different  due  to  smaller  reductions  in  design  scope. 
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number,  and  timing  of  both  design  and  integration  errors.  Generally,  early  design 
errors  are  expected  to  be  reduced,  but  later  integration  errors  may  increase  due  to 
evolving  standards.  Errors  that  are  discovered  and  addressed  during  an  acquisition 
program  are  not  as  problematic  as  those  that  remain  after  the  product  has  been  put 
into  service.  Undiscovered  and  released  errors  are  problematic  because  they  can 
severely  increase  operations,  maintenance,  and  upgrade  costs. 

The  model  was  used  to  simulate  the  number  of  undiscovered  errors  in 
released  work  without  and  with  open  systems.  Figure  6  shows  the  evolution  of  the 
number  of  undiscovered  and  released  errors  as  a  percent  of  the  program  scope.  In 
general,  the  number  of  released  errors  increases  as  work  is  completed,  until  the  next 
development  phase  begins  receiving  development  work,  finding  errors,  and  returning 
them  to  upstream  phases  for  resolution. 


Undiscovered 
and  Released 
Errors  as 
Percent  of 
Scope 


Time  (Week) 


Figure  6.  Undiscovered  Problems  in  Evolutionary  Acquisition 
without  (Line  1)  and  with  (Line  2)  Open  Systems 

Figure  6  shows  that  the  simulated  project  with  open  systems  generates  and 
fails  to  find  and  resolve  more  errors  before  release.  To  further  investigate  this,  the 
errors  were  disaggregated  into  design  errors  and  integration  errors — based  on  the 
assumption  that  errors  in  the  early  development  phases  of  each  block  (requirements 
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and  technology  development  and  advanced  development)  are  primarily  design 
errors,  and  errors  in  manufacturing  and  testing  are  primarily  integration  errors. 
Figure  7  shows  the  undiscovered  and  released  design  errors  as  a  percent  of  scope 
with  and  without  open  systems,  and  Figure  8  shows  the  undiscovered  and  released 
integration  errors  as  a  percent  of  scope  with  and  without  open  systems.  Note  that 
the  vertical  scale  in  Figure  8  (0-20%)  is  four  times  larger  than  the  vertical  scale  in 
Figure  7  (0-5%)  for  clarity. 


Undiscovered 
and  Released 
Design  Errors 
as  Percent  of 
Scope 


Figure  7.  Undiscovered  and  Released  Design  Errors 
in  Evolutionary  Acquisition 
without  (Line  1)  and  with  (Line  2)  Open  Systems 

The  differences  in  the  timing  of  when  design  errors  are  generated,  discovered 
and  resolved,  or  missed  and  released  is  primarily  due  to  the  faster  development  with 
open  systems.  More  importantly,  the  total  percent  of  design  errors  at  the  completion 
of  the  program  is  nearly  the  same  for  the  two  programs.  This  suggests  that  the 
important  impacts  of  open  systems  on  evolutionary  acquisition  may  not  lie  in  design 
errors. 
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Figure  8.  Undiscovered  Integration  Errors  in  Evolutionary  Acquisition 
without  (Line  1)  and  with  (Line  2)  Open  Systems 

There  are  at  least  two  important  differences  between  the  number  of 
undiscovered  and  released  design  errors  (Figure  7)  and  the  number  of  undiscovered 
and  released  integration  errors  (Figure  8).  First,  the  programs  generated  and  failed 
to  resolve  three  to  four  times  as  many  integration  errors  than  design  errors.  This 
suggests  that  PMs  using  open  systems  must  address  integration  issues  if  they  wish 
to  succeed.  This  finding  also  supports  the  importance  of  the  shift  from  design  to 
integration  identified  by  other  investigators.  Second,  the  program  with  open  systems 
generated  at  least  25%  more  integration  errors  than  the  program  without  open 
systems  (3+%  more  than  13%).  This  difference  in  integration  errors  explains 
essentially  the  entire  difference  in  total  undiscovered  and  released  errors  (Figure  6). 

In  summary,  the  simulation  results  show  that  open  systems  can  interact  with 
evolutionary  acquisition  to  improve  the  timing  of  products  (Figure  5),  reduce 
development  costs,  and  increase  the  number  of  undiscovered  and  released 
integration  errors  (Figures  6-8).  This  suggests  that  open  systems  and  evolutionary 
acquisition  can  interact  to  improve  schedule  and  cost  performance,  but  that 
these  benefits  may  come  at  the  cost  of  increased  risk  of  high  operations, 
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maintenance,  and  upgrade  costs  when  the  integration  errors  are  eventuaiiy 
discovered  and  must  be  resoived. 
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Implications  for  Evolutionary  Acquisition  Practice 
with  Open  Systems 


The  identification  of  impacts  of  open  systems  on  evolutionary  acquisition 
programs  and  the  simulation  results  carry  potentially  valuable  implications  for 
acquisition  program  managers. 

Shifting  the  Types  and  Amounts  of  Risk 

Adding  open  systems  to  evolutionary  acquisition  shifts  the  program 
management  focus  from  design  to  standards  and  integration.  This  impacts  when  the 
program  accepts  and  must  manage  different  types  and  amounts  of  risk.  Open 
systems  reduce  design  risks  by  designing  components,  subsystems,  and  systems  to 
be  consistent  with  established  standards.  Component  design  risk  is  also  reduced,  as 
an  OS  approach  uses  pre-designed  and  pre-tested  components  that  have  been 
designed  and  tested  to  established  standards.  Open  systems  may  increase  other 
risks,  however.  Standards-selection  and  change  risks  are  increased  because 
programs  using  open  systems  are  dependent  on  standards  more  than  programs 
using  customized  designs;  OS  also  have  little  influence  over  the  evolution  of  those 
standards.  Integration  risks  may  increase  significantly  as  standards  change  over  the 
product  lifecycle,  and  new  standards  may  not  be  compatible  with  the  current  design 
of  products.  Different  types  of  skills  are  needed  to  manage  different  types  of  risk.  For 
example,  detailed  component-design  risk  management  requires  technical  expertise 
for  design  review  and  component  testing,  but  integration  risk  management  requires 
a  broader,  systems  understanding  of  the  product  and  how  subsystems  work  together 
to  fulfill  requirements.  Acquisition  programs  using  open  systems  need  a  different  set 
of  risk-management  skills  than  programs  not  using  open  systems.  Less-detailed 
technical  expertise  will  likely  be  needed,  and  more  integration  and  systems  expertise 
will  be  needed.  .  If  open  systems  are  integrated  into  evolutionary  acquisition 
(which  repeats  the  development  process  over  multiple  blocks),  acquisition 
programs  will  require  significant  and  extended  integration  and  systems 
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expertise.  This  will  also  change  the  skill  sets  needed  by  the  DoD  acquisition 
workforce. 

A  Temporal  Shift  in  Program  Risks 

Design  risks  occur  relatively  early  in  programs  and  product  lifecycles, 
whereas  integration  risks  occur  relatively  late.  Therefore,  the  use  of  open  systems 
will  shift  program  risk  to  emerge  later  in  projects.  The  simulations  support  this  result 
with  the  increase  in  the  number  of  undiscovered  and  released  integration  errors  with 
open  systems.  If  costs  follow  risk,  this  may  result  in  lower  development  costs  due  to 
lower  design  risk,  but  higher  operating,  maintenance,  and  upgrade  costs  due  to 
higher  integration  risk.  Figure  9  describes  the  relative  costs  in  a  product  lifecycle. 
Integration  of  OS  into  EA  may  reduce  Research  and  Development  costs  when 
programs  can  capture  design  benefits,  but  may  increase  Operating  and  Support 
costs  when  integration  and  evolving  standards  risks  may  increase  costs.  The  sizes 
of  these  cost  changes  are  uncertain,  but  the  potential  for  early  reductions  in  cost  and 
later  increases  in  cost  are  real. 
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Figure  9.  Relative  Costs  during  a  Product  Lifecycle 

(Defense  Acquisition  University,  2004,  November,  p.  43) 
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By  stretching  acquisition  across  muitipie  biocks,  evoiutionary 
acquisition  may  accentuate  the  impacts  of  a  temporai  shift  in  program  risk. 
Therefore,  if  using  open  systems  causes  this  temporai  shift  in  risks,  then 
programs  integrating  open  systems  and  evoiutionary  acquisition  may 
experience  an  increase  in  the  reiative  size  of  product  costs  during  use. 

Trading  Design  Obsoiescence  for  Integration  Obsoiescence 

Traditional  acquisition  processes  commit  programs  to  customized  designs 
and,  therefore,  bear  significant  design  obsolescence  risk  when  threats  and 
technologies  evolve  away  from  the  design.  An  open  systems  approach  can  reduce 
that  risk  by  allowing  the  use  of  more  plug-and-play  components  that  can  be  replaced 
with  improved  components  that  meet  the  chosen  standard.  However,  by  using  open 
systems,  a  program  must  also  commit  to  one  or  more  standards  early  in 
development  and,  therefore,  bear  significant  standards  obsolescence  risk  if  and  as 
standards  evolve  away  from  the  needs  of  the  program  and  as  integration  problems 
increase.  Evoiutionary  acquisition’s  need  for  integration  across  muitipie 
deveiopment  biocks  can  increase  the  impact  of  open  systems  on 
obsoiescence  risk.  Adding  open  systems  to  evoiutionary  acquisition  may 
cause  programs  to  trade  away  design  risk  for  increased  integration  risk. 
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Conclusions 


The  current  work  has  extended  and  expanded  the  descriptions  of  the  impacts 
of  using  open  systems  and  evolutionary  acquisition  together  on  development 
processes  and  management.  We  then  mapped  those  impacts  into  a  computer 
simulation  model  and  used  that  model  to  investigate  how  open  systems  and 
evolutionary  acquisition  interact.  Results  include  that  the  changes  required  to 
implement  open  systems  in  evolutionary  acquisition  significantly  impact  development 
processes  and  management,  particularly  scopes  of  design,  standards,  and 
integration  work,  the  generation  of  different  types  of  problems,  and  the  timing  of  the 
discovery  of  problems.  The  shift  from  a  focus  on  design  to  a  focus  on  integration 
was  found  to  be  particularly  important.  Simulation  reinforced  the  potential  for  open 
systems  to  accelerate  acquisition  and  revealed  a  potentially  important  distinction 
between  design  and  integration  errors  in  explaining  the  impacts  of  required  changes. 
Implications  for  practice  included  shifts  in  the  type  and  timing  of  risks  due  to  open 
systems  use  and  the  possibility  of  trading  design  obsolescence  for  integration 
obsolescence. 

This  research  has  contributed  to  the  understanding  of  open  systems  and 
evolutionary  acquisition  in  several  ways.  The  work  improved  the  description  and 
specification  of  impacts  of  acquisition  policy  on  acquisition  practice.  The  work  also 
used  dynamic  computer  simulation  to  model  and  investigate  open  systems  and  to 
model  evolutionary  acquisition  and  open  systems  together,  both  for  the  first  time  to 
our  knowledge.  The  results  of  the  simulation  reinforced  several  suggested  impacts 
of  open  systems  and  provided  additional  causal  rational  behind  why  suggested 
impacts  may  occur.  These  rationales  were  the  basis  of  potential  implications  for  the 
evolutionary  acquisition  practice  with  open  systems.  The  reasoning  provided  based 
on  the  computer  simulation  can  be  used  to  extend  and  deepen  decision-makers’ 
understanding  of  open  systems  and  evolutionary  acquisition  and  the  design  of 
program  processes  and  management. 
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Future  researchers  can  improve  and  extend  the  work  described  here  by 
gathering  additional  data  about  the  use  of  open  systems  with  evolutionary 
acquisition  in  practice  and,  in  so  doing,  testing  the  existence  and  importance  of 
suggested  impacts.  The  similarity  of  the  model  and,  thereby,  confidence  in  results 
can  be  improved  by  using  additional  acquisition  projects  that  use  both  evolutionary 
acquisition  and  open  systems.^  Finally,  additional  recommendations  for  practice  can 
be  developed  based  upon  the  model  developed  here  and  elsewhere.  These 
investigations  can  further  develop  the  understanding  of  how  to  effectively  integrate 
open  systems  and  evolutionary  acquisition  and,  consequently,  improve  the  systems 
and  products  provided  to  warfighters. 


^  The  authors  are  eurrently  working  with  a  large  navy  aequisition  projeet  to  do  this. 
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Appendix  A.  Mapping  Specific  Influences  of  Open 
Systems  onto  Evolutionary  Acquisition  Programs’ 
Processes 


The  researchers  estimated  the  impact  of  each  specific,  identified  and 
described  influence  on  the  scope  of  work,  rework  fraction,  and  quality  assurance 
(QA)  effectiveness.  They  measured  the  scope  of  work  by  the  number  of  equal-sized 
work  packages  that  must  be  completed  in  a  development  phase.  They  measured  the 
rework  fraction  with  the  percent  of  those  work  packages  that  require  changes;  this 
measurement  reflects  the  number  of  problems  that  are  created  in  a  development 
phase.  They  measured  the  QA  effectiveness  with  the  fraction  of  the  work  packages 
needing  rework  that  are  discovered  to  need  rework.  Although  no  known  data  is 
available  as  a  basis  for  the  estimated  changes,  order  of  magnitude  estimates  that 
are  in  a  reasonable  rank  order  of  size  are  adequate  because  of  the  preliminary 
nature  of  the  study.  To  facilitate  mapping  of  the  specific  influences  listed  in  the  text 
to  model  changes,  the  researchers  listed  the  specific  influences  after  the  individual 
impacts  on  each  model  parameter  in  parenthesis. 

Table  3.  Detailed  Estimate  of  Changes  in  Evolutionary  Acquisition  Processes 

to  Reflect  Open  Systems 


Program  Block 

Scope 

Rework 

QA 

and  Phase 

of  Work 

Fraction 

Effectiveness 

DEVELOPMENT  BLOCK  1 

Requirements 

+1+1+5  (1,4,10) 

0 

0 

Develop  Technology 

+1+1+1-20+1+1(1,2,3,5,8,9) 

0 

-5  -5  (7a,7b) 

Advanced  Development 

+1-20+1+1  (4,5,8,9) 

-10+5(5,6) 

-5  -5  (7a, 7b) 

Manufacturing 

+1  +1(4,9) 

+5  (6) 

+10  -5  (7a,7b) 

Testing  by  Users 

+im 

0 

-5  (7b) 

Net  Change  in  Base  Case 

-22 

0 

-20 

DEVELOPMENT  BLOCK  2 

Requirements 

+1  (4) 

0 

0 

Develop  Technology 

+1+1  -20+1+1  (3,4,5,8,9) 

0 

-5  (7a) 

Advanced  Developoment 

+1-20+1+1  (4,5,8,9) 

-10  +5  +5(5,6,8) 

-5  (7a) 

Manufacturing 

+1  +1(4,9) 

+5  +5  (6,8) 

+10  (7a) 

Testing  by  Users 

+im 

0 

0 

Net  Change  in  Base  Case 

-29 

+10 

0 

DEVELOPMENT  BLOCK  3 

Requirements 

+1  (4) 

0 

0 

Develop  Technology 

+1+1-20+1+1  (3,4,5,8,9) 

0 

-5  +5  (7a,7b) 

Advanced  Development 

+1-20+1  +1(4,5,8,9) 

-10  +5+10  (5,6,8)-5  +5  (7a,7b) 

Manufacturing 

+1+1  (4,9) 

+5  +10  (6,8) 

+10+5  (7a,7b) 

Testing  by  Users 

+1  (4) 

0 

+5  (7b) 

Net  Change  in  Base  Case 

-29 

+20 

+20 
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2003  -  2008  Sponsored  Research  Topics 


Acquisition  Management 

■  Software  Requirements  for  OA 

■  Managing  Services  Supply  Chain 
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